Online Unsupervised Multilingual Acoustic Model Adaptation for Nonnative Asr

Automatic speech recognition (ASR) is currently one of the main research interests in computer science. Hence, many ASR systems are available in the market. Yet, the performance of speech and language recognition systems is poor on nonnative speech. The challenge for nonnative speech recognition is to maximize the accuracy of a speech recognition system when only a small amount of nonnative data is available. Recent studies on nonnative speech recognition were focus on supervised context in which spoken languages (L2) and speakers’ mother tongue languages (L1) are known in advance. In this paper, we want to study the adaptation approach of nonnative speech in which both L1 and L2 are unknown in advance. Such new approach is called online-unsupervised multilingual acoustic model adaptation. Thus, “unsupervised” means we don’t know in advance the nonnative speech utterance (it’s L1 and L2); and “online” means the adaptation is made during the decoding. Thus, the proposed approach decomposes into two stages. The first stage, contained language observer module, aims to recover the linguistic information (spoken languages and the origins of the speakers) of the unknown speech utterances to be decoded. The second stage is to adapt the multilingual acoustic model based on knowledge provided by language observer module. It is clear that the multilingual acoustic model must contain the acoustic units of L2 and L1. In this study, we report on the acoustic model adaptation for improving the recognition of nonnative speech in English, French and Vietnamese, spoken by speakers of different origins. Results degradation around 7% of baseline systems’ phone error rates (PERs) obtained from the experiments demonstrate the feasibility of the method.